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decision-making are considered, based on a conceptual scheme for 
comparing information-based change efforts. Based on a student 
outcomes project, eight brief case studies of public colleges 
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efforts, the following types of variables are important; the' 
structure and process of institutional dncision-making; the use of 
information in decision-making; and strategies employed to induce 
change basad on assessment information. Specific variables include: 
institutional type and mission, school size, array of programs, locus 
of decision-making; resource flexibility, the quality and extent of 
available information, pressure for information use by top 
administrators, utilization of unit resources, and the integration of 
the change effort with ongoing activities. Appended are interview 
questions used in the National Center for Educational Management 
Systems Student Outcomes Project, along with a description of the 
format for repotting campus activities. Eight pages of references 
conclude the report. (SW) 
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Effectiveness Information and Institutional Change: 
An Exploratory Analysis 



Introduction 

A primary objective of all college and university administrators is to increase 
the effectiveness of undergraduate instruction. Recently, calls for 
improvement in this arena are many, and include such national reports as NIE*s 
Involvement in Learning (1984), AAC's Integrity in the College Curriculum 
(1985), and SREB's Access to Quality Undergraduate Education (1985). A 
fundamental assumption in all these reports, however, is that explicit 
Information about student performance in attaining instructional objectives is 
a crucial starting point for academic improvement. Following this logic, a 
number of states and accrediting bodies have recently changer^ their 
accountability and resource allocation mechanisms (see, for example, Bogue and 
Brown 1982, Folger 1984, Ewell 1985, SACS 1984, Thrash 1984). 

Past experience, however, suggests that most colleges and universities face 
formidable organizational obstacles in attempting to implement 
information-based improvements in instruction and decisionmaking. The reasons 
for this are several, and involve two kinds of difficulties. A first set of 
problems arises from the place of undergraduate education in most colleges and 
unversities, and from the uncertain legitimacy of educational assessment and 
measurement. A second set of problems is more general ^ and concerns the nature 
of information Itself as an ingrediant of decisionmaking in higher education 
settings. Both sets of problems have a considerable bearing on the degree to 
which information-based change strategies, as are currently being proposed, can 
in fact be successfully implemented. 
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Four Obstacles to Change 



In order to analyze the success of a change effort, it is first necessary to 
Identify what stands In its way. Both past research and accumulated 
institutional experience suggest four basic obstacles to using effectiveness 
information as a lever for improvement in undergraduate instruction. 

First, institutions often lack a clear commitment to high quality undergraduate 
education (NIE 1984). Institutional mission statements, particularly for 
public institutions, are broadly constructed, and undergraduate instruction is 
only one of many identified priorities (Wallace 1985). In the case of large, 
multi-purpose research universities, this phenomenon is expected, and is 
perhaps appropriate. But in smaller, regional public institutions where 
undergraduate education arguahiy should be paramount, it also lacks priority 
(Birnbaum 1985). As a result, even where informition on instructional 
effectiveness is available, it is not clear that it will automatically be used 
to improve instruction. 

Secondly, at most institutions there is a tendency toward fragmented 
respoirsibllity for undergraduate education. This is partici'larly true in the 
area of general education, where there is usually no single office or 
individual responsible for coordinating and monitoring actions to improve 
student success. In part, this weakness is a result of typically strong 
departmental structures and their associated disciplinary cultures (Clark 
1983). General education usually occupies a low priority in relation to the 
academic major, responsibility for which is centered in individual departments 
or academic units (Gamson and Associates 1984). Even when administrative 
responsibility for general education is located in a specified unit (for 



example, a general college or a division of general studies) this unit is often 
regarded as "second class" and is shunned by discipline faculty. 

The typical division of labor between academic affairs and student affairs 
functions is also problematic for systematic change. Faculty, particularly in 
larger institutions, tend to abdicate responsibility for wholistic intellectual 
development in the belief that this is being professionally handled elsewhere 
(Astin 1985). This is despite considerable evidence that high quality 
faculty/student interaction has a broad range of positive developmental effects 
on students (for exampie, Feldman and Newcomb 1969, Astin 1977, and Terenzini 
and Pascarella 1977). Student service professionals, at the same time, have 
tended ♦o develop services and activities in isolation from other parts of the 
institution. As a result, programs and policies of academic and student 
affairs offices are rarely discussed and developed in concert. Effectiveness 
information that speaks to the impact of the instructional environment as a 
whole thus will often have nowhere to go. 

Thirdly, concrete incentives for using effectiveness information in any 
decision forum are generally lacking. Academic budgeting processes remain 
largely based on instructional volume rather than on the outcomes of 
instruction (Bowen 1978). This is partly a product of public sector funding 
practices that allocate the bulk of an institution's resources on the basis of 
enrollment-driven funding formulas (Brinkman 1984, Jones 1985). Unit-level 
administrators see substantial rewards for increased enrollment and teaching 
volume, and they structure their own budgetmaking practices accordingly. Most 
private institutions exhibit similar behavior: units that generate substantial 
income are rewarded, and income is in most cases tied to enrollment. In such a 



climate, Information on effectiveness is a luxury rather than a decisionmaking 
necessity. 



Finally, the nature of Information about instructional effectiveness Itself 
creates difficulties of application In most college and university settings 
(Ewell 1982), A first difficulty is purely cultural: many of the presumed 
outcomes of higher education are held to be in principle unmeasurable, and 
attempts to gather and use effectiveness information are resisted purely on 
this basis. A second problem is disagreement about what to measure. The 
intended outcomes of higher education are remarkably diverse, and vary markedly 
across institutional types (Pace 1975, Pascarella 1985). A third problem is 
the fact that information on student learning and development really is more 
complex than the kinds of information routinely used by academic administrators 
in making day-to-day decisions (Pace 1985). Because such information is 
collected indirectly through a variey of measurement Instruments, rather than 
being directly observed, and because the technology of educational measurement 
often involves the use of techniques that are not immediately "face-valid" to 
decisionmakers, the difficulties of translating measurement data into usable 
Information can be formidable (Jones 1982). Perhaps most Importantly, 
information on educational outcomes rarely directly tells decisionmakers what 
actions to take. Unlike the types of information that decisionmakers are 
accustomed to handling, it is difficult to direcly link a particular finding 
from an effectiveness study to a particular institutional policy or program 
that needs changing (Ewell 1984a). Such data more often will highlight the 
presence of a problem, will provide context for a range of decisions, or will 
serve as a stimulant for discussion or action (Ewell and Chaffee 1984). 



The Broader Role of Information In Decisionmaking 
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These largely operational points raise a range of broader questions about the 
role of concrete information in the higher education decisionmaking process. 
Considerable past research points to the fact that the utilization of 
evaluation Information of all kinds can vary considerably across different 
organizational settings and across different types of decisionmakers. In 
higher education, for example, past research has demonstrated that the role 
played by information may depend upon such factors as the formal organization 
of decisionmaking (Coleman 1972), the political positions of those generating 
and using the information (Baldridge 1971), or the general cultura of 
decisionmaking (Chaffee 1983). In addition, the disciplinary backgrounds of 
college and university decisionmakers, particularly academic decisionmakers, 
may profoundly affect the perception and use of explicit assessment 
information. Studies that examine disciplinary background and decisionmaking 
style, for example, suggest that decisionmakers whose academic training was in 
different fields may perceive and use quantitative data quite differently 
(Mitroff 1982). At the same time, studies that investigate the role of 
individual cognitive styles in the perception and use of decision Information 
suggest similar variability (for example, McKenney and Keen 1974). 

A second set of Issues surrounds the manner in which concrete Information 
actually enters the decision process. Most formal treatments of the role of 
information in decisionmaking are based on a rational model of organizational 
process. According to this model, those who make decisions seek Information in 
order to clarify the probable consequences of alternative future courses of 
action- Possession of information, while it cannot make the decision, can 
reduce uncertainty about which alternatives show the greatest potential benefit 
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while incurring the least cost (Raiffa 1968). Many constraints operate on this 
notion in practice, however, and tend to highlight the "non-rational" uses of 
information in decisionmaking. For example, Ewell and Chaffee (1984) identify 
four such constraints in college and university settings— (1) constraints of 
incomplete information, (2) political constraints, (3) constraints of 
organizational culture, and (4) constraints imposed by the need for the 
organization to take unambiguous action. Some of the "non-rational" uses of 
information that they observed included the role of explicit performance data 
in focusing attention on a neglected issue, in "selling" a decision previously 
arrived at on other grounds, and in inducing concrete action. 

Many of these uses are explored in the wider literature on organizational 
behavior. Information may, for example, serve as a "signal" to outside 
constituents that an organization is rationally (and therefore appropriately) 
administered (Feldman and March 1981). Negative information in particular can 
serve to focus administrator attention— in itself a scarce organizational 
resource (March 1982, Braskamp and Brown 1980). Finally, information may be 
most effective in its role of promoting agreement and consensus so that 
consistent organizational action is possible. For example Brunsson (1982) 
documents several cases where the primary role of information was to mobilize 
the organization to take an action— any action— rather than to help leaders 
decide among a set of posed alternatives. 

Many of these dynamics are also observable in the history of explicit 
information utilization projects in higher education. For example, in one such 
project the utilization of institutional planning and management information 
depended upon perceived linkages between planning information and visible, 
unit-level problems (Baldridge and Tierney 1979). In another project that 
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attempted to apply Information on student learning and development to 
curriculum decisions, utilization was often blocked by differences among 
disciplinary cultures, and by the perceived symbolic role of the Information 
provided (Astin 1976). 

A third set of Issues concerns the manner In which information Is collected and 
communicated to decisionmakers. Past research on knowledge util ligation 
suggests that the successful application of knowledge requires the simultaneous 
presence of a number of conditions. First, the Information must have a visible 
bearing on an actual perceived problem. Information utilization in this sense 
never takes place in the abstract. Secondly, there must be a constant and 
consistent dialogue between those who gather and provide information end those 
who must use it. With regard to the first condition, Rogers and Shoemaker 
(1971) describe five stages through which a decision to adopt an innovation 
must pas . Similarly, Lazarsfeld and Reitz (1975) describe a cyclical "map" of 
knowledge utilization that stresses the manner in which decision information is 
rooted in and must make constant reference to an explicit problematic 
situation. The close ties between utilization and concrete problems is also a 
strong theme in the evaluation literature. For example, Patton (1978) and 
Coleman (1972) stress that decisionmakers are willina to make use of evaluation 
data that is seen as problem-related, even if they are aware that it is of 
questionable accuracy. 

Moreover, the ways in which information is actually presented to decisionmakers 
can deeply Influence whether or not it is used. Evidence from the evaluation 
literature, for example, suggests that the language in which a particular 
problem is expressed, and the visual forms in which data are presented can have 
a considerable Impact on its credibility (Newman, Brown and Braskamp 1980, 
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Stufflebeam 1971, Anderson and Ball 1978, Havelock 1973). Futhermore, as 
Boland (1980) stresses, available Information defines the character of 
decisionmaking itself. The kinds of information collected, maintained and 
visibly disseminated play an important part in defining the official "language" 
of an organization. 

In higher education, these issues have been discussed by those interested in 
the rational use of information in policymaking (Schmidtlein 1977, Jones 1982), 
and by those ''nterested in the intereaction between modes of information 
presentation and cognitive style (Hackman 1983). In examining the process of 
using information on student outcomes, for example, Kinnick notes both 
••organizational" and ••technical*' obstacles to utilization (1985). Among the 
former are lack of organizational access to information, lack of an appropriate 
organizational framework for integrating information from different sources and 
of different kinds, limited incentives to seek out and use information, and 
lack of established channels of communications between those who develop, 
manage, and use information. Among the "technical" factors identified are 
excessive bulk in reporting, organizing presentations around discrete data 
collection efforts rather than around issues or problems, lack of data 
integrity, lack of face validity, inadequate timeliness, and limited data 
interpretability. Strategies for overcoming some of these obstacles in the 
realm of data communication included graphic presentation, iterative release of 
information, and redundant data presentation strategies. 

A Conceptual Scheme for Comparing Information-Based Change Efforts 

As the above review suggests, accounting for the success or failure of 
information-based instructional improvement efforts is bound to be complex. 
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Making sense of It demands proceeding on conceptual and observational fronts 
simultaneously. The balance of this section presents a detailed conceptual 
framework for beginning to diagnose such efforts. The section that follows 
attempts to apply this approach to a range of actual cases where assessment 
information was used as a conscious change agent. 

A major difficulty in evaluating such efforts to date, however, is their 
relative scarcity, and the fact that individual efforts have grown up in 
isolation from one another. Three past multi-institutional projects stressing 
the use of student outcomes information are worth noting— an eight-institution 
project on retention data undertaken by the Higher Education Research Institute 
at UCLA (Kemarer, Baldridge and Green 1982), a seven-institution project on 
student outcomes information sponsored by the W. K. Kellogg Foundation through 
the National Center for Higher Education Management Systems (Ewell 1984b), and 
an eight-institution FIPSE project on the use of "value-added" information in 
institutional decisionmaking (Astin 1984). In addition, several 
institution-specific studies of the use of student assessment data have become 
available. Among these are studies of small private colleges (for example 
Mentkowski and Doherty 1984, Mentkcwski and Loacker 1985), studies of major 
research universities (for example Banta 1985a, Banta 1985b, Wallace 1985), and 
studies of regional public comprehensive universities (for example McClain and 
Krueger 1985, McClain 1984, Dumont and Troelstrup 1981). Evidence provided by 
these examples suggests that the presence of assessment information in the 
institutional decisiomaking process can have a major role in the direction of 
policy, but that general understanding of how the utilization process works and 
what makes it effective remains limited. This evidence does, however, provide 
an initial basis for constructing a conceptual scheme for comparing, 
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contrasting, and judging the success of Information-based instructional 
improvement efforts. 



Three Layers of Context for Information-Based Change Efforts 

One way of beginning the construction of such a scheme is to recall several 
successive layers of obstacles noted in the literature review above. A first 
layer has to do with issues unique to student assessment information, and 
unique to the problem area of improving undergraduate instructional 
effectiveness. Such issues include the particular problems of measurement and 
communication associated with information on student learning and development, 
and the organizational difficulties that on most campuses surround the delivery 
or and responsibility for undergraduate general education. A second type of 
problem, in contrast, concerns information utilization in general, and how 
information of all kinds is collected, disseminated, and used in 
decisionmaking. Such issues Include the organizational locus and control of 
information, the kinds of information regularly collected and communicated, and 
the kinds of decision situations in which information is regularly used. A 
final layer of problems has to do with the actual character of decisionmaking. 
Issues here include its relative centralization or decentralization, the kinds 
of issue-areas present for decisionmaking, the "rationality" the decision 
process, and its patterns of regularity and formality. Each of these layers of 
Issues, of course, is related, and each tends to encompass and constrain the 
one within it (Figure 1). Together, however, they constitute and structure a 
functioning "environment" within which individual campus-based change efforts 
must operate. 
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Figure 1 

Environments for Information-Based Change E forts 
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This simple conception raises a number of basic questions about the process of 
Information-Induced change. These questions can, In turn, serve as a basis for 
organizing and comparing quite different bodies of institutional experience; 

1. What are the primary variables that operate within and that have a 
demonstrable effect on each "layer" of environment? What major 
factors, for example, condition the way in which all decisions appear 
to be made at the institution? What determines or conditions the 
availability of decision information of differing kinds, and the ways 
in which such information is used in different settings? Finally, 
what especially conditions or determines the use and availability of 
1nformati<^n on Instructional effectiveness—a particular species of 
decision information? 

2. What kinds of interactions occur among these different "layers" of 
environment? How do these interactions, if they are present, shape 
the processes that occur at each stage of the information utilization 
process? Are there, for example, institutional "syndromes" that link 
particular patterns of decisionmaking with particular types of 
informa*,ion use? If such patterns or syndromes exist, it may be 
useful to broadly d stinguish among a few broad classes of 
institutions within which quite different approaches to 
information-based change ar'^ appropriate. 

3. Given different environments, what kinds of explicit change strategies 
seem to be most effective in what kinds of situations? This, of 
course, is the ultimate policy question, and can only be answered once 
a great deal of information on different institutional experiences is 
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collected. But case study work can begin to chart the directions for 
more systematic future work in this arena. 



These three broad questions frame a considerable research agenda, and the 
ingredients for undertaking only the initial stages are currently available. 
Making a start now, however, may be of considerable benefit in guiding future 
efforts. 

A Catalogue of Key Variables 

Given a multi-contextual functioning environment for information-based change 
efforts, a first analytical task is to begin to explicitly identify the 
principle factors that affect utilization efforts within each layer of context. 
To the extent that such factors can be identified and compared, particular 
institutional experiences can be placed in their proper environmental setting, 
and inappropriate generalizations to other settings can potentially be avoided. 
The following catalogue of key variables represents a first attempt at 
accomplishing this task. Three basic classes of variables are identified— (1) 
variables affecting the structure and process of institutional decisionmaking, 
(2) variables affecting the use of Information in decisionmaking , and (3) 
variables describing the kinds of strategies employed to induce change ba^ed on 
assessment information. Variables included under each heading are briefly 
discussed below. 

A. Institutio nal ConU (t; The Structure and Process of Institutional 
D ecisionmaking 

Variables included under this heading correspond to the broad 
decisionmaking efivironment present at the institution. The results of 
otherwise similar improvement activities may be vastly different given the 
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different contexts set by these Important groups of "control" variables. 
Such general environmental variables include the following: 

1. Institutional "Demographics" 

- The Type and Mission of the Institution . This includes the 
institucion's basic type (community college, regional, comprehensive, 
research center, etc.) as well as any distinctive elements of mission 
which the institution may stress (agricultural/technical, regional 
service, etc.). 

- The Size of the Institution . Size may be described in many ways, but 
for current purposes, institutions differ primarily in the number of 
studrnts served, the number of distinct programs offered, the 
complexity of the administrative structure, and the level of 
available resources. 

- The Institution's Array of Programs . This incl'ides the number of 
programs offered, the levels at which they are offered, and the 
number of distinct fields in which programs are offered. 

- Student Body Characteristics . This includes most of she primary 
demographics of the student body (age, employment, minority status, 
residence), their academic aptitudes, their academic goals, anl their 
enrollment characteristics (full-time, evening, degree-seeking, 
etc.). 

- Relative Stability . This refers to the degree to which the above 
characteristics have remained relatively stable over time. 
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Centralization of Decisionmaking , These variables refer to the actual 
process of decisionmaking on campus. Is the decisionmaking process for 
the most part a participatory and decentralized process, or is it 
concentrated at the upper- administrative levels? Or does this depend 
upon the type of decision? In general, two kinds of factors need to be 
distinguished: 

- The ''Mission Solidarity" of the Institution . This refers to the 
degree to which all decisionmakers at the institution feel that they 
are trying to accomplish the same thing. To the extent that this is 
present, actual decisionmaking may be highly decentralized in 
practice, with little risk that incongruent or inconsistent decisions 
will be made. 

- The Locus of Decisionmaking . This refers to the actual physical 
location of different kinds of decisions. In this context the formal 
decisionmaking system must be distinguished from the informa l 
settings in which it operates. In addition, the degree to which 
decisions are delegated may be different in different issue areas. 

Institutional Flexibility . These variables refer to the degree to 
which the institution is able to take independent action to soIvj its 
problems and to improve its programs. In general, three factors shoild 
be distinguished: 

- Decision Latitude . This refers to the relative ability of the 
institution to take action independent of its external governing 
bodies. How is the institution constrained in what it can do because 



of policies or procedures requiring approval or clearance from higher 
authorities? 

- Resource Flexibility , This refers to the institution's relative 
ability to reallocate resources (financial, human and physical) to 
meet new needs. If the institution is financially constrained, 
resource flexibility will probably be at a minimum. At the same 
time, substantial resource inflexibility may still exist even if 
resources themselves are relatively plentiful, 

- Orientation Toward Risk and Innovation , This refers to the history 
and style of the institution in dealing with risk or uncertainty. Is 
the general pattern to consciously innovate, or to attempt to control 
the environment in order to preserve relatively established ways of 
doing things? Will new initiatives be seen primarily as a threat or 
as an opportunity? 

Use of Information in Decisionmaking 

Variables included under this heading consist of those that influence the 
inner two "rings" of environmental context for information-based change 
efforts. Essentially, the same set of questions can be asked about the 
institution's general use of information in decisionmaking, and about the 
use of information on instructional effectiveness. But it is important to 
explicitly ask these questions at both levels. Appropriate change 
strategies may well be quite different it the challenge is to move a 
particular campus toward information-centered decisions in general, rather 
than to include instructional assessment information in an existing 
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decision process that already recognizes the legitimacy of 
Information-based decisionmaking. Several factors are Important here: 

1. The Quality and Extent of Available Information . Use of Information in 
decisionmaking will first depend upon the range and adequacy of the 
information available. Both for information in general and for 
information on student learning and development, the particular types 
of Information available need to be identified and assessed in terms of 
their frequency of collection and in terms of their general quality 
(validity, reliability, applicability to institutional problems). 

2. The Locus and Control of Information . The availability of decision 
information is also a function of who collects it, of who keeps it, and 
of what policies exist for maintaining and disseminating it. For both 
general decision information and for student outcomes information, the 
following dimensions need to be considered: 

- Information Centralization . This refers to the degree to which 
information collection is centrally organized or is dispersed among 
the various units in the Institution. What particular areas of 
decision Information are centralized? How uneven is the information 
collection process among units across the institution? Do the 
content of unit information bases and central information bases 
differ substantially and in what areas? 

- Information Dissemination/Availability . This refers to the relative 
accessibility of Information once collected. Is Information tightly 
held by those who collect it, or is it widely shared with others in 
the institution? Are these patterns different for different units 
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and for different kinds of Information? What units have explicit 
responsibility for information dissemination and how do they execute 
this responsibility? 

Receptivity of Decisionmakers , These variables govern the "listener" 
side of the data communication problem. At the broadest level, two 
kinds of considerations are relevant: 

• The General Cognitive Style of Decisionmakers . This refers to 
decisionmakers* general approach to the use of information. 
Information may be extensively used to explore different decision 
alternatives and to weigh their relative merits. It may be used to 
justify previously taken decisions or as a weapon in the 
organizational bargaining process. Finally, it may be tightly or 
loosely held once acquired. 

- The Use of Specific Kinds of Information . Although they may share a 
general attitude toward information, particular decisionmakers may 
have quite different attitudes toward the utility of particular kinds 
of information. Despite a positive general altitude toward 
Information, some may find data on instructional effectiveness to be 
entirely irrelevant to the management problem. Others may be 
positively attracted to information about what students are learning, 
despite a generally non-rational approach to using information in 
decisionmaking. 

Pressure for Information Use by Top Administrators . Independent of any 
of the above, the use of information in decisionmaking will be affected 
by the resolution of top administrators tc enforce information use. 
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Aside from the degree or level of pressure present in the situation, 
three other factors should be considered: 



- Nature of Incentive . This refers to whether the incentives provided 
for information use are positive or negative. Are units rewarded for 
using information in decision processes or are they directed to use 
information regardless of their own feelings as to its 
appropriateness? 

- Delegation of Authority . This refers to the degree to which 
monitoring and enforcing information use is kept as a top-level 
function or is formally or informally delegated. 

- Degree of Fol low-Up . This refers to the way compliance or 
non-compliance in using information is handled by the Institution. 
Are unit responses to information-use initiatives carefully evaluated 
and followed up, or is the process a more informal and evolutionary 
one? 

Change Strategy Dimensions 

While the variables outlined in vhe previous section condition the success 
of particular information-based change efforts, those outlined below 
constitute the actual structure of such initiatives. The first set 
comprises environmental factors which are relatively fixed and to which 
particular change strategies must adapt. The second set, on the other 
hand, constitutes a range of choices. Which choices are most effective and 
most appropriate given a particular array of environmental conditions is, 
of course, the primary policy question for most administrators. 



Characteristics of the Unit Acting as "Change Agent" 



The choice of where in the institution to locate responsibility for a 
change project is fundamental. Inevitably, actions undertaken by this 
unit in support of the project will be seen as actions by the unit 
itself. They will also be influenced by the relative position of the 
unit in the decision structure, its communication links with other 
units, and its perceived goals and function within the institutional 
community. Several dimensions of unit context are worth explicit 
consideration: 

- Proximity to the Locus of Decision . This refers to the distance 
between the unit primarily responsible for using information to 
initiate change and the locus of decisionmaking in the institution. 
Is the unit responsible itself a decision unit, or is it a support 
unit? If a decision unit, what Icinds of decisions does it directly 
influence, participate in, or determine? Wh/^t kinds of decisions are 
excluded from its domain? 

- Unit Resources Available ^ This refers to the particular resources 
that the unit can bring to bear directly upon the project. These 
include particular skills and experience, particular sources of data, 
particular types of access to communications channels, and so on. 
These also include the kinds of external resources that the unit can 
mobilize to support the effort, such as grant assistance, consulting, 
or high visibility for the initiative. 

- Unit Resources Utilized . This refers to the degree to which the 
above resources are actually utilized in support of the change 




effort. To what extent do other unit ohjectlves consume resources 
which might have been devoted to accomplishing improvement or 
information use objectives? Or, on the other hand, to what extent 
does attainment of related unit objectives help also to accomplish 
the objectives of the change initiative? 

- Stability of Primary Actors . This refers to the degree to which the 
same people remain involved in key positions throughout the entire 
course of the project. In what ways does personnel turnover, either 
in the project itself or in key administrative positions in the 
institution, influence the development of improvement efforts? 

Target Units/Groups 

The choice here is essentially at whom to direct the thrust of the 
information utilization effort. Several dimensions are important here: 

- Scope of Target . This refers to the degree to which the initiative 
is aimed at influencing all units in the institution or only a narrow 
range of such units. Is the effort, for example, aimed primarily at 
academic or at student service units? At what level? 

- Specificity of Objective . This refers to the type of change aimed at 
particular target groups. Is the objective to change particular, 
identified actions or policies, to change an operating style, or to 
change broad attitudes toward the use and value of outcomes 
information? 



Integration of the Change Effort with Ongoing Activities 



The choice here is how completely to distinguish institutional 
improvement activities from regular, ongoing institutional activities. 
On the one hand, a distinct set of activities may heighten 
consciousness of the improvement effort on campus. On the other hand, 
changes made as a result of such an initiative may turn out to be 
surface changes— not penetrating to the level of actual organizational 
practice. Again, several dimensions of this choice are relevant: 

- Distinctiveness of Objective ^ This refers to the degree to which the 
objective of the initiative is already an established institutional 
or unit objective. If the objective is to impro./e existing ongoing 
processes through inclusion of effectiveness information, related 
activities may be fairly difficult to distinguish from other 
activities directed toward similar ends. If, on the other hand, the 
objective is a new one, the activities undertaken may be easily 
distinguishable from other institutional activities. This will also 
be true for objectives for which no explicit unit responsibility is 
currently assigned. 

- Distinctiveness of Image . In contrast to the above, this refers to 
the perception of the change effort on campus as a distinct, separate 
activity. Is the initiative seen as a way to accomplish an 
identified, distinct objective? Or is it rather seen as a source of 
support for an internal unit or campus initiative? 

- Relation to Established Lines of Authority . This refers to the 
degree to which the organization of the change effort follows or cuts 
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across the institution's established patterns of authority and 
organization. How much cooperation among distinct administrative 
areas is required and achieved? What is the function of special 
bodies and committees that cut across regular lines of authority? 
What informal mechanisms for promoting inter-unit cooperation are 
available and employed? 

Problem Orientation 

The choice here is the degree to which the objectives of the change 
effort are tied to an explicit, identifiable campus issue or problem. 
Linking the goals of the initiative with particular campus problems may 
have the substantial advantages of harnessing the perceived urgency of 
the problem and of providing an explicit focus for particular 
activities. On the other hand, linking project objectives to an 
explicit problem may involve the substantial cost of losing momentum 
toward more general data use once the problem selected goes away. To 
the degree a particular data-use initiative Is problem directed, it is 
useful to distinguish several dimensions of the problem: 

- Permanence of Problem . This refers to the degree to which the 
problem as posed can be solved within the terms and time- span of the 
project. Most problems will have elements with at least some 
short-term solutions. 

- Sourci; of Problem . This refers to the degree to which the problem is 
externally imposed upon the institution. If the source of the 
problem is external, the change effort may obtain a great deal of 
leverage from that fact. A major question, then, will he how to 



internalize this leverage when external pressure is no longer a 
factor. 

An Initial "Database" for Institutional Comparison 
Oocumenting Patterns of Institutional Experience 

As noted above, there are relatively few well documented current cases of 
institutions attempting to implement explicit instructional improvement 
programs on the basis of effectiveness information. One of the few sets of 
institutions available is a group of 22 colleges and universities participating 
in the operations and dissemination ph'-ses of a four-year national project on 
the use of student outcomes information in institutional program planning and 
decisionmaking (Ewell 1984b). Beginning in 1981, seven public institutions 
participated in the project— each institution undertaking an explicit 
problem-centered effort to use existing information on student learning and 
development. This core group of institutions was joined in 1984 by an 
additional fifteen public colleges and universities each undertaking a similar 
effort. 

Although intended as a consulting-based institutional change project, NCHEMS 
staff collected considerable research data in the course of the project. Each 
year, both local project personnel and NCHEMS consultants prepared extensive 
debriefing reports on campus activities using a defined common format (see 
Appendix A). Questions in the debriefing report covered not only practical 
problems encountered, but also included many items that the literature on 
information utilization suggested might be important contextual variables. In 
addition, at the conclusion of the three-year effort, faculty and 
administrators at the seven original sites were interviewed using a formal, 
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open-ended interview schedule (Appendix A). Ten to twelve interviews were 
conducted on each campus and the results were compiled for later research use. 
The same procedures ^re being used to document the progress of the fifteen 
dissef.ii nation institutions whose information-utilization projects ire s'.ill in 
process. The result of these documentation efforts is a rich initial 
"database" for investigating patterns of success among such institutional 
change efforts. The twenty-two cases covered, all public institutions, range 
in size and type from a small, rural community college with an enrollment of 
less than 2000, to a large, "flagship" research university with an enrollment 
of over 25,000. While the number of institutions is admittedly few, reviewing 
some of their experiences In the light of an explicit conceptual framework 
should prove iielpful to more formal investigations as greater numbers of cases 
become available. 

The balance of this section consists of eight brief case reviews drawn from 
this initial "database"— each of which illustrates a particular pattern of 
interaction between campus environment and the successful utilization of 
assessment informa\:ion to induce change. The major features of each 
institution discussed are docuinented in Chart 1. Because information 
utilization was in each case conditioned by complex patterns of institutional 
culture and local political conditions, the individual identities of each of 
tne institutions discussed are protected by pseudonyms. 

While the interactions among the many tictors conditioning Information use are 
complex, an initial attempt at classifying each case discussed was made using 
the conceptual scheme presented in the previous section. These results are 
presented in Chart 2. The pattern of classifications for <*ach case in Chart 2 
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Chart 1 

Features of Case Institutions 



Name 


Type 


Location 


FTE Enrollment 


Frontier State University 


Publ ic Research 


Rural 


10,100 


Midstate University 


Public Research 


Urban/Suburban 


15,500 


Sunkist Communitv Collpap 


Public Two-Ypar 


>J 1 L/U M 


14 snn 


^Msta Cormunity College 


Public Two-Year 


Suburban 


10,300 


Heartland State University 


Public Comprehensive 


Rural 


6,700 


Coldspring State University 


Public Comprehensive 


Rural/Suburban 


15,900 


Southern State University 


Public Research 


Urban/Suburban 


19,500 


Central University 


Public Research 


Urban/Suburban 


30,200 
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Chart 2 

Conceptual "Profiles" of Eight Case Institutions 
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1. Institutional -Demographics" 
(see Chart 1) 


















2. Centralization of Decisionmaking 

* Mission Solidarity 

* Locus of Decisionmaking 


High 
Low 


Med. 
Low 


High 
High 


High 
Med. 


Med. 
High 


Low 
Med. 


Hed. 

High/ Low* 


Med. 
^ Low 


3. Institutional Flexibility 

- Decision Latitude 

- Resource Flexibility 

«- Orientation Toward Innovation 


High 
Ked. 

High 


Low 
Low 
Low 


High 
High 
Med. 


High 
Low 

High 


Med. 
Low 

High 


Med. 

Low 

Med. 


Lew 

Med. 

Low 


Med. 
Med. 
Med. 


Use of Information in Decisionmaking: 


















1. Quality/Extent of Information 


Med. 


High 


Low 


High 


Med. 


Low 


High 


High 


£. locus/Control of Information 
- Information Centralization 
* Information Availability 


Low 
Med. 


High 
Low 


High 
Low 


High 
High 


High 
Med. 


Low 
Low 


High 
Low 


Med. 

High 


3. Receptivity of Decisionmakers 


Med. 


Low 


Med. 


High 


Med. 


Low 


Low 


Med. 


4. Pressure for Use by Top Administrators 
* Nature of Incentive 

- Delegation of Authority 

- Degree of FollOM-Up 


Positive 

High 

Low 


Negative 
Low 

High 


Negative 

Low 

High 


Positive 

High 

High 


Mixed 

Low 

Med. 


Negative 
Med . 
Low 


Negative 

LOW 

Low 


Positive 
High/ Low 
High 


Change Strategy Dimensions: 


















1« Unit "Change-Agent** Characteristics 

- Proximity to Locus of Decision 

- Unit Resources Available 

- Unit Resources Utilized 

- Stability of Primary Actors 


Med. 
Med. 
Med. 

High 


Med. 
Low 

High 
High 


High 
Low 
High 
High 


High 
High 
High 
High 


High 
Low 
High 
High 


High 

Low 

Low 

High 


Low 
Med. 

High 
Low 


High 
High 
High 
High 


2« Target Units/Groups 

- Scope of Target 

- Specificity of Objective 


Broad 
Low 


Broad 
Low 


Broad 
High 


Broad 
Low 


Broad 
Med. 


Broad 
High 


Narrow 

High 


Med. 

High 


3. Integration with Ongoing Activities 

- Distinctiveness of Objective 

- Distinctiveness of Image 

- Relation to Established ^^^^^^ of 
Authority 


Low 
Med. 

Low 


Low 
Low 

High 


Low 
Low 

High 


Low 
High 

Hign 


High 
High 

n lijii 


Med. 
High 

ill .lu 


High 
Med. 

1 . 


Med. 

High 


4. Problem Orientation 

^•^^rmanence of Problem 
cn ir^urce of Problem 


Low 

Internal 


Med. 

Internal 


Low 

External 


Low 

Internal 


High 
Internal 


Med. 

External 


Low 

External 


Med. 

External 
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serves as a useful organizing principle for both discussing individual cases, 
and for noting some broad interactions among the various variable dimensions. 

So me Cases 

Patterns of experiences in Chart 2, of course, can only begin to document the 
complexities of interaction between campus environment and information 
utilization that occurred at each institution. As a result, the following 
brief reviews of particular cases attempt to highlight the dominant pattern of 
relationships and to note the ways in which particular operational "syndromes" 
were present. Cases are presented in pairs in order to emphasize a particular 
contrast in approach, and the institutions in each pair are roughly comparable 
in size, type, and mission orientation. Fach case presentation will 
approximately follow the logic of Chart 2 by presenting (1) the basic 
"demographics" of the institution, (2) the general role of information in 
decisionmaking, (3) the kinds of "change strategies" employed by the 
institution, and (4) the kinds of changes actually accomplished and the impact 
of information in the change process. All variables in 1-3 are reported as of 
the beginning of the change effort. In some cases, the pattern of these 
variables changed as a result of the effort itself. 

Information Utilization and Unit Entrepreneurship 

Frontier State University is a wdium-sized residential land-grant institution 
serving a total student population of approximately 10,000 (8500 undergraduate 
and 1500 graduate students). The array of programs is considerable and 
includes such areas as engineering, agriculture, architecture, veterinary 
medicine, and forestry as well as business md the traditional array of liberal 
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arts disciplines. Students are largely traditional, full-time, live on campus, 
and are drawn predominantly from within the state. 

Mission distinctiveness and mission solidarity are considerable at Frontier 
State; most faculty and administrators see the institution as a unique 
environnent with its own lifestyle and campus culture. Most are at the 
institution by choice, and would feel uncomfortable in a different environment. 
Most decisionmaking is informal —either entrepreneurial or consensual. 
Individual units have considerable decision latitude to establish and implement 
their own policies. At the same time, much informal communication and 
coordination between units themselves and with campus administration occurs 
informally through a network or assistant deans and such "information brokers" 
as the Director of Institutional Research. 

Partly as a result of this pattern, information use is mixed at Frontier State. 
The quality of centralized information resources— for example, the registration 
database— Is extremely low, and is currently under review. But most 
departments and units tend to keep a good deal of their own information. At 
the same time, individual unit decisionmakers vary considerably in their 
approach to and use of information. In the Engineering School, for example, 
the collection and use of local information about students is considerable. 
The President is also positively oriented toward information, but more as a 
device to package anH cpii decisions than as a major aid to decisionmaking. 

The essence of the University's change strategy was a combination of positive 
incentive and unit initiative. Two efforts were central. First, the Office of 
Institutional Research— the locus of the effort— attempted to locate, document, 
and make available for use a wide range of student-outcomes studies 
accomplished across the campus. Secondly, small amounts of incentive funds 
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were made available to individual units in the form of "mini-grants" to 
undertake local 1nfor,;iation-use projects. Twelve such projects were funded, 
the most successful of which involved the use of student success data to 
prepare improved recruitment materials that stressed the need for prospective 
students to take an appropriate pattern of college preparatory courses while in 
high school . 

Long-term results of the effort were mixed. Reliance on unit initiative meant 
that (1) many units did not become involved, and (2) many that did decided not 
to pursue the effort after encountering initial difficulties. Nevertheless, 
some useful short-term products resulted. Because the effort was driven by 
local problems and positive incentives, and because it depended upon local 
motivation, those units that elected to proceed with an Initiative generally 
did so successfully. But no formal or ongoing mechanism to ensure that 
information could again be used emerged. The utilization effort tended to stop 
with the perceived "solution" of the problem motivating Involvement. 

Effectiveness Information In a Formalized Process 

Midstate University is one of several major research universities in a 
multi-campus state system. It currently enrolls about 15,000 students, 
approximately 12,000 of whom are undergraduates, and maintains a strong 
traditional array of graduate and research programs. The undergraduate student 
population (residential and traditional) is recruited selectively, and is drawn 
primarily from the major city In the state. 

Like Frontier State, Midstate has a strong sense of institutional 
distinctiveness— particularly in the institution's research mission— and a 
self-image of high quality is an important element of institutional identity. 



27 



ERJC " 34 



ERIC 



Unlike its counterpart, however, a recent retrencfiment produced a strong 
presidency and a formal, highly participatory institutional planning process 
for setting institutional funding priorities. Hard, quantitative information 
on unit costs, markets, and enrollments plays a prominent part in the planning 
process. The use of information has been encouraged by the institution's 
current President— a quantitative social scientist with a strong personal 
interest in and affinity for policy data. It is also supported by a high 
quality institutional research staff that conduct a wide range of policy 
studies. The results of such studies, however, have not in general been widely 
disseminated nor extensively used by unit-level administrators. 

In contrast to Frontier State, the change effort at Midstate was both 
centralized and formal. Beginning in 1982, results of a number of student 
performance studies (most notably a survey of alumni that noted graduate school 
attendance and persistence, professional placement and development, and 
retrospective ratings on several dimensions of the quality of instruction 
received) were included along with enrollment and cost criteria, in the annual 
planning/budgeting process. As part of this process, units were required to 
speak to their strengths and weaknesses as revealed by these data, and to 
propose plans for the coming year accordingly. 

Because the use of data on performance was centrally sanctioned and 
incorporated in a formal decision process, it was guaranteed attention. Few 
units chose not to cooperate. But because the effort was seen as external to 

the unitb, anu ma^ nOt pif Oblcni-bGS^d, fCal utilization of the <r\fnrmAf<:,i% 

beyond "compliance" tended initially to be limited. Units initially used 
information effectively only as they surfaced ways the information might help 
them deal. with a particular, local issue. Unlike Frontier State, however, a 
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permanent mechanism for utilization was established to support such efforts on 
an ongoing basis. Perhaps as important, individual unit heads reported that 
the inclusion of such information in the formal process served as an 
unambiguous "signal" that the central aiimini strati on valued effective 
undergraduate instruction. 

"Top-Down" Utilization in a Community College Setting 

Sunkist Community College is a three-campus community college serving some 
14,000 headcount students In a major urban area. The college's programs are 
wide-ranging, and include a substantial baccalaureate transfer component, 
community service programming, and a strongly supported array of one and 
two-year occupational programs. Students are a typical community college 
commuting population, although Sunkist has a higher proportion of traditionally 
aged full-time students than many other institutions of its type. The college 
is supported by a combination of state aricl local tax revenues, and the 
President is largely concerned with external relations, and the maintenance of 
support from the Board. Day-to-day operations at the institution are handled 
by a central office staffed by three young, active Vice-Presidents, and the 
three campus Provosts. 

Decisionmaking at Sunkist tends to be highly centralized. The mechanics of the 
budgetmaking and curricular design processes are closely held by the central 
office, and many decisions are made informally among the six primary actors. 
Information has been Increasingly valued In the decisionmaking process, but 
oblditriri9 yOOu "iPiformCiticr hcis been a problsf!!. Reglst^^t'^o'^ d^t^^hA^pc wptp in 
the past not well maintained, and timely, reliable information on enrollments 
was often not available. Recently, through the initiative of the 
Vice-Presidents and the manage^nent team, this has begun to change. Substantial 
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Investments have been made in improving data completeness and in developing new 
computer software to track student progress. A registration/admissions "task 
force" composed primarily of administrators recently designed an interactive 
MIS for registering, placing and counseling students. Plans are also in place 
for an automated degree audit to be implemented as part of this system. 

The use of instructional effectiveness Information at Sunkist was stimulated by 
an announcement that the state community college system would soon embark on a 
program of systematic program review. The management team at Sunkist saw this 
event as an excellent opportunity to develop a local program review effort --a 
process that might then be adopted by the state as a model. External 
consultants were hired to help design the process, which included a two-tiered 
review effort and a substantial amount of external data gathering. Among the 
data collected were extensive Interviews with area employers that emphasized 
the need for and performance of Sunkist students In various occupations. The 
process adopted also Involved creation of a faculty review committee empowered 
to make recommendations on program di recti on--the first such body in the 
Institution's history. 

Programs chosen for review In the Initial year included many that the 
management team felt were weak and needed redirection or closing. Three such 
programs— Hospitality Management, Fashion Merchandising, and Social Work--were 
thought to be particularly weak. Available data allowed discussions of the 
future of these programs to be undertaken in a manner quite different from 
former practice at Sunkist: faculty were given the chance to make their best 
rsktiP for A nroar;»m. and then weigh the available evidence about its 
effectiveness. One program was terminated and another redirected. A third, 
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however, was supported after previously unarticulated evidence of effectiveness 
was compiled and discussed. 

Because of its "top-down" culture, information utilization at Sunkist was first 
seen as a managpr^ent initiative. Faculty were reluctant to participate in the 
process and felt inadequate in doing so. Indeed, the initiative was 
seen— partly correctly— as an attempt by administration to kill some 
undesirable programs. Because faculty were included in the effort, and because 
information was openly shared, however, the process was gradually seen as 
legitimate. Adoption of the Sunkist model by the state also helped enhance the 
program's credibility. As a result, a small but central core group of faculty 
was "socialized" into the decisionmaking process. Many of that group are 
currently at work on a much larger initiative— the use of student tracking and 
test score data +o evaluate curriculum effectiveness. 

"Bottom-Up" Utilization in a Community College Setting 

Demographically similar to Sunkist, Vista Community College is a multi-campus 
institution in a growing suburb of a major city. Although its clientele is 
slightly older and more part-time than that of its counterpart, its array of 
programs, size, and funding mechanisms are all quite similar. Institutional 
culture and decisionmaking styles, however, are quite distinctive at Vista. 
Led for the past decade by a dynamic president. Vista is continually 
experimenting with innovative management processes— most of them highly 
participatory. Over the years, for example. Vista has used focus groups, 
"quality circles," and other such techniques to raise issues about the 
institution's future and current performance. The pace, however, can be 
frenetic; indeed some participants feel that an atmosphere of constant change 
and endless dialogue may get in the way of attaining real effectiveness. 
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Vista has Invested heavily in management information, and has a well supported 
computing and information facility staffed by an open and able staff. Unlike 
most institutions, manegemer.t information, administrative computing, and 
institutional research functions are combined under a single director. As a 
result, management information is plentiful, and its availability is 
facilitated by a wide-ranging computer communications network throughout the 
institution. 

Systematic utilization of effectiveness information at Vista began with 
formation of an institution-wide Student Success Task Force some five years 
ago. Like many such ad-hoc bodies at Vista, the Task Force was large (some 40 
members) and was drawn from all parts of the institution. Also like many 
previous efforts, the objectives and focus of the effort were initially rather 
difficult to pin down. Formed as a result of genuine concern about student 
progress throughout the institution, the charge of the committee was simply too 
large to get anything accomplished. Early discussions, however, eventually 
pointed to the need for additional concrete information about student 
intentions, and about student success after completion of their programs. 
Subcommittees of the Task Force then concentrated on each of these efforts. 

At the same time. Vista's Vice President worked in parallel with the 
institution's management team in the routine use of outcomes information. By 
organizing problem- solving meetings around particular bodies of data, broken 
down by division and program, the Vice President slowly changed the language of 
decisionmaking to an information-laden language. Rather than requiring that 
particular bodies of information be used in particular ways, he encouraged and 
subtly rewarded day-to-day information use by individual members of the 
management team. As a result, division chairs and program directors soon began 
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sharing techniques with each other, and making necessary improvprneri\s in both 
information content and in the process of communicating information. An active 
"Information constituency" could at this institution be adequately supported 
because of earlier Investments in high quality equipment and management 
Information personnel. In other settings, it might not have worked. 

An Incremental App**oach to Information Utilization 

Heartland State University is a regional comprehensive state university located 
in a rural area in a midwestern state. It currently enrolls about 6500 
students--the vast majority of them undergraduates--all of whom are 
traditionally aged, residential, and full-time. Reflecting its origins as a 
state teacher's college, programs are offered in education and business, as 
well as the full range of traditional liberal arts disciplines. Headed by a 
long-serving President, the Institution's administrative structure is simple 
and stresses "low overhead" operations. Reporting to the President is a single 
Dean of Instruction, supported by a small staff, who administers six broad 
academic divisions. Department chairs are half-time appointments. Academic 
decisionmaking tends to be concentrated in the Dean's council, which although a 
collegial body, primarily concerns itself with implementing the President's 
policies. 

Some twelve years ago. Heartland began a program of testing designed to 
demonstrate the national competitiveness— and hence the credibil ity— of its 
degree earners. All students are tested on graduation using nationally normed 
achievement tests in their major field of study. Earlier in their programs, 
stifde^t^ toctoH in npnpral prfiiratiftn, aUo usina nationally available 
standardized achievement tests. Results are compared with norms, and are used 
in building a case for undergraduate instructional effectiveness--both to 
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prospective students and to the legislature. Largely designed and initiated by 
the President— himself a professional e<lucator— administration of the 
evaluation prograin Is centei'ed In the Dean's office. 



When the program was begun. Its focus was largely external. In several 
successive years, for example, Heartland was able to acquire additional funding 
for quality Improvement from the legislature on the basis of demonstrated past 
and promised future student performance. But sf"«;ral deficiencies in 
curriculum uncovered by the testing program were brought to the faculty'*', 
attention and were addressed. Most notable of these was a 
math-across-the-curriculum initiative begun as a direct coiisequpnce of low math 
performance on thr ^^eneral education tests. Recently, however, familiarity 
with the data has led faculty to make increasing use of it to revise curricula 
and academic policies. Moreover, the testing program itself has developed 
incrementally since its inception. New Instruments have been periodically 
added to the program, and the coordination of a student survey effort improved. 
Institutional planning and budget-building documents have gradually been 
modified to incorporate data as they are gathered, and utilization of 
information at the unit level has been aided by gradual development of a simple 
set of reports on various dimensions of student performance. 

Four years ago, interviews with typical faculty members at Heartland revealed 
the testing and evaluation program to be an initiative perceived largely as an 
externally directed effort. Few people at the institution beyond its top 
leadership were actively involved. Today the situation is different. A clear 
majority of faculty are aware of and Involved in the data utilization effort. 
The most recent Instance of ongoing data use, for example, is provided by 
Heartland's science faculty, who are currently engaged in revising Introductory 
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course sequences in the basic sciences guided by several years of general 
education test score results. A persistent but gradual approach to using 
infor-iation has resulted in a strong base of faculty support, largely because 
various approaches were given sufficient time to demonstrate their potential 
and because precipitous judgments about programs were not immediately made on 
the basis of fragmented evidence. 

A Case of Rapid Implementation 

Coldspring State University is a developing regional comprehensive college 
serving approximately 15,000 students. About a third of the student body is in 
full*time residence, with the balance commuting from two major urban areas 
within driving distance of the campus. Like Heartland, Coldspring State is a 
former teachers college, but unlike Heartland, it has grown rapidly over the 
past decade as the region's population has grown. The array of programs 
offered, however, has remained relatively stable— a set of liberal arts core 
disciplines with strong business and education schools that enroll a majority 
of the current student body. 

Oecisionmiiking at Coldspring State is relatively decentralized, and individual 
school Deans have considerable latitude to manage their own units. At the same 
time, information is badly scattered, and there are few central foci for its 
collection arid interpretation. Partly this is due to rapid expansion: 
"planning" at Coldspring State still means physical plant development, and top 
administrative attention has been directed largely tow?.rd the design and 
financing of new buildings. Nor is this attention misplaced; Coldspring has 
the highest classroom utilization rates in the state, and facilities are 
palpably inadequate to handle continued expansion. Concentrating on physical 
plant development and the information needed to support it, however, means that 
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other Information functions have suffered. Reliable numbers on student 
progress are difficult to get and their implications remain undeveloped. Nor 
has top administration up to now valued information beyond that needed to 
manage day-to-day operations and that needed to answer question:, from the board 
and legislature. Enrollment has never been a problem, and, because funding is 
largely enrollment driven, instructional effectiveness has not been a visible 
issue. 

The impetus for change came with a new President three years ago. The current 
President is politically well connected, and from the beginning sought an 
opportunity to demonstrate innovation. Like administrators at Sunkist, he saw 
an opportunity in contemplated state board policies on measuring instructional 
outcomes. Seeking to be the lead campus in a possible statewide program, the 
President called for a systematic testing program aimed at end-of-year 
sophomore students. All such students were to be tested in math and writing 
skills using standardized, nationally normed test instruments as a condition of 
further progress. 

The proposed testing program was attached to a very short timeframe. A pilot 
test was scheduled for an initial class of sophomores, but ^est results were to 
be actually used in promotion decisions within a ^ear. Departmental faculty in 
English and Math were usked to identify appropriate instruments in conjunction 
with the Director of the University's Testing Center, but the decision on which 
tests to use was taken quickly and without broad consultation or input. More 
importantly, there was little opportunity to investigate the enrollment or 
curricular consequences of a testing program whose impact was uncertain. Early 
indications suggested, for example, that as many as two-thirds of current 
sophomores (and practically all currently enrolled minority sophomores* woulc^ 
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perform below a proposed cutoff score on the math examination. Public 
announcements of the testing program, however, were already in place. The 
University wa? thus faced with an unfortunate choice between potentially 
substantial enrollment losses in the upper division, and setting a cutoff score 
so low that the program itself would be a political embarrassment. Moreover, 
early results indicated that students performed quite differently on the math 
examination depending on the class and section taken. Clearly the University 
was not delivering a uniform "product" in its approach to mathematics general 
education, and until it did, there were few equitable grounds on the basis of 
which it could demand equivalent performance of all its students. Fortunately, 
the writing po**tion of the testing program was not subject to these 
difficulties. As a result, the University was able to proceed with this 
portion of the program to fulfill political promises while it rethought the 
math situation. A hasty attempt to implement a complex program, however, had 
ultimately cost the University both time and internal political good will. 

A Problem-Centered Utilization Effort 

Southern State University is a major land-grant institution located in the 
Southeast. Southern State enrolls about 19,000 students (16,000 of whom are 
undergraduates) and supports over a hundred academic programs varying from 
Textiles to Engineering. Like many institutions of its size and type. Southern 
State is decentrali/.ed and fragmented. Eight distinct colleges offer academic 
programs— each with its own policies, procedures, and admissions requirements. 
Student Service functions are quite distinct from academic decisionmaking, and 
are more centrally directed. Central direction, however, flows from the top, 
from the Vice Chancellor through his associates to individual unit directors. 
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Rarely do line personnel from each of these units routinely work together to 
solve coinnon problems. 

Southern State collects a considerable anount of Information on student 
progress and performance through its Office of Institutional Research. These 
data, however, are not traditionally used in the decisionmaking process; indeed 
many schwl deans have essentially duplicated studies already accomplished 
centrally—not knowing that student tracking data are already available. The 
budgeting process tends to be based on individual unit bargaining; no formal 
institution-wide planning or program review efforts are in place. 

The stimulus for systematic information use at Southern State came in 1981 with 
the University's need to comply with a Federal Consent Decree mandating 
achievement of minimum minority enrollment goals by 1986. At first, the 
University concentrated on recruitment to meet this goal, but it was soon 
apparent that a more comprehensive approach to minority enrollment management 
was needed. An opportunity for concentrating efforts occurred with the 
availability of external grant cpport. Under the auspices of a nationally 
supported project, an Southern State committee was formed to use available data 
on student performance to develop comprehensive minority enrollment management 
policies. Because the effort began in Student Affairs, members of the 
Committee were at first largely confined to Student Affairs personnel. Data on 
retention, recruitment, placement, and academic performance were considered in 
turn, and their implications developed. Considerable coordination among 
student affairs programs resulted from this process. Also, considerable use of 
information was made by individual unit managers in developing and evaluating 
the effectiveness of their own programs. But utilization of this information 
was rare beyond the Student Affairs area. Perceived as a "non-academic" 
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initiative, individual academic deans were either not aware of the -effort, or 
ignored it. 



This picture changed somewhat with a second set of initiatives— development of 
an integrated academic skills program centered in the Provost's office, and 
with the creation of a set of faculty minority coordinators to work in each 
college. Representatives of both initiatives were included in the committee, 
and its mandate was broadened. With minority success rates growing and with 
external grant support and visibility coming to an end, however, the 
utilization effort gradually became more diffuse. Looked at as a distinct 
"project" rather than as an ongoing effort, the change process gradually lost 
focus. With the accession of a new Chancellor and a consequent new set of 
Priorities, campus administration was reorganized. Although many of the 
concrete improvements made as a result oT the effort remain, few visible 
reminders the change initiative at Southern State are visible today. 

Building an "Information Culture" 

Central University is a land-grant research institution serving ? total of 
30,000 students (including about 24,000 undergraduates) located in a mid-sized 
state. The University offers over 120 programs through 14 distinct colleges 
and schools, and routinely secures major research grants from outside agencies. 
Its undergraduate student population is competitively recruited from throughout 
the state and region, and is of traditional age, is full-time, and residential. 
Like most large research institutions, administration is decentralized. 
Recently, however, central initiatives such as strategic planning and progrfrn 
review have been initiated under the leadership of the Executive Vice 
Chancellor. Also like many large research universities, information resources 
have in the past been scattered. Institutional Research collects a range of 

Er|c "46 



enrollment and financial data, individual schools and departments conduct 
studies of their own, and an independent learning suppcrt and testing unit 
collects considerable information on student learning and development. As a 
result, information utilization has in the past been both sporadic and 
unit-specific. 

The impetus for information-based change efforts at Central came with 
development 6f a statewide incentive funding program established six years ago. 
In return for collecting and effectively using student performance Information, 
institutions in the state system could qualify for considerable additional 
funding. At first, the motivation for collecting such information was purely 
instrumental, anJ information "utilization" was seen largely as a compliance 
exercise. Top administration at Central, however, quickly saw the external 
funding mechanism as an opportunity to build and utilize an Instructional 
evaluation database that would provide excellent support for developing 
strategic planning and program review initiatives. 

Coordination of data collection and utilization efforts was from the outset 
seen as essential at the University, and this responsibility was given to the 
learning support and testing unit. As a first step in building an integrated 
program, staff of this unit interviewed each academic dean to determine data 
needs and availability. Three working task forces Involving faculty and 
administrators were then formed around different types of performance data. 
Each was charged with determining the kinds of Instruments and data collection 
processes that were available, appropriate, and operationally feasible. By the 
end of a year, recommendations from the task forces were available, and a data 
collection design developed. To begin testing the design, funds were made 
available by the achninistration for individual departments to experiment with 
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collecting and using infomation. Eventually, fourteen such pilot projects 
were underta!^ an , ranging from the use of student satisfaction data in faculty 
evaluation to the use of cognitive testing in the major field to improve 
curriculum. ing lessons from the pilot project, a full-scale instructional 
evaluation program began two years after the effort was first conceived. 

Hallmarks of the current program are central data collection and local 
utilization of the resulting information. Program staff compile data 
"profiles" of each college and school for presentation to the dean and for 
discussion with department chairs and faculty. They also actively promote 
discussion of findings and their implications as part of the annual program 
review process. Because of the perceived importance of statewide incentive 
funding to the University (now over $3 million), instructional evaluation is 
seen as an important and legitimate activity. But because of its development 
through a participatory process and the active data-based dialogues promoted by 
both learning research staff and the formal program review process, many 
department faculties also see the utility of these data to improve their own 
curricula and programs. As their examples convince others on the faculty that 
such activities have a local, departmental utility, an information constituency 
has developed throughout the campus. Through careful planning, appropriate 
involvement, and an adequate time horizon for development, what could have been 
a compliance exercise became instead an institution-wide program with a 
permanent and ongoing impact. 

Some Implications 

Although each of the cases described above is quite distinctive, together they 
illustrate a number of common principles. First, each pair of cases highlights 
a conmon tension inherent in such efforts. An institution's choice about how 
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^1 to resolve each tension has much to do with the likely course its utilization 
effort will take in practice. Secondly, the cases as a body illustrate a 
number of themes present in the development of any such effort. Each of these 
topics will be briefly explored below. 

Institutional Change St rategy as Effective Choicemaking 

In attempting to put into practice information-based change, many strategies 
are potentially available. The strengths and weaknesses of each are inherent 
in the approach chosen, and also result from the appropriateness of the choice 
given the larger campus culture within which the change strategy must operate. 
Four dimensions of choice are illustrated in the cases just presented, and each 
represents a combination of the kinds of variables embedded in the more general 
conceptual scheme described earlier. 

1- Centralization vs . D ecentralization . Essentially this is a choice of 
the degree to which local initiative is encouraged in the utilization 
effort. As seen in the case of Frontier State, considerable 
involvement can be built throug^^ use of an entrepreneurial strategy, 
but at the cost of change being extremely uneven across the 
institution. Efforts such as those at Midstate will produce a more 
uniform result, but may take considerably more time to implement; 
worse still, such efforts may fail because there are no local 
incentives to sustain them. Good compromise positions are illustrated 
by Central University and Vista Community College, both of which used 
centrally coordinated processes, but allowed considerable unit 
initiative in carrying them out. 
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Problem-Centered vs. Process Centered Efforts . The literature on 
knowldege utilization stresses that information is most effective as 
an ingrediant in decisionmaking when it can be explicitly brought to 
bear on a shared and highly visible problem. Soithern State's 
minority student success effort, Sunkist's attempt to remedy the 
deficiencies of a few already identified academic programs, and both 
Central State's and Coldspring State's responses to proposed statewide 
initiatives are all examples of the ways local problems were used to 
ground a more general utilization effort. These cases also show that 
use of a bounded problem can be a mixed blessing. Southern State's 
change efforts essentially ceased with the attainment of minority 
recruitment goals. Similarly, unit utilization of information at 
Frontier State rarely was generalized beyond the particular unit-level 
problem attracting attention. But where the problem at hand could be 
used as a "stepping-stone" to creating a wider utilization effort, it 
was successful in mobilizing attention. This was clearly the case at 
Central University and at Sunkist Community College. 

Top-Down vs. Bottom-Up Management . Neither of these approaches 
appeared dominant in the abstract. Rather, the success of change 
efforts depended on the degree to which their management pattern was 
congruent with the more general management style of the institution. 
At Sunkist, a "top-down" approach was expected by the faculty and 
worked well. At '.'ista and Frontier State, the opposite was the case. 
At Central University end Coldspring State, however, the change 
efforts represented a more "top-down" approach than was typical on 
these large, decentralized campuses. Their differences in success 
were due largely to the relative amounts of time allowed for the 
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effort~1n the case of Central University a considerable length of 
time, and for Coldspring State an extremely short one. 

4. Long-term vs. Short-term Approaches . Change efforts with careful 

planning and plenty of time for execution generally worked better than 
those that were implemented on a short time-frame. Central University 
and Heartland both involved plenty of opportunity for pilot testing of 
policies before implementation, and allowed the interests of all 
parties to be heard and to be potentially incorporated into policy. 
Coldspring State's efforts did not enjoy this luxury. But short 
time-frame efforts also worked in the right settings. Sunkist's 
program review effort was developed quite quickly, and Southern State 
was able to field programs within a year. In the first case this was 
due to the ability of a centralized administration to move quickly and 
legitimately to create and implement a new program. In the second, it 
was due to the percieved urgency of a common institutional problem. 
Neither of the^e mitigating conditions was present at Coldspring 
State. 

Some Common Themes of Information Use 

While each of the cases discussed represents a different pattern of choices 
with regard to the four tensions outlined above, all illustrate the efficacy of 
sdme common themes of information use. A first theme is one of gradually 
building participation through changing patterns of individual administrative 
behavior. In all eight cases, faculty and most administrators were inif*ally 
indifferent to or actively opposed to gathering and using performance 
information. A first step in changing this situation involved altering the 
parameters within which administrators operated—either through direct changes 
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in the incentive or constraint structure, or through changes in the official 
language of institutional discourse. For example, Midstate's use of a 
well-established planning process for institutional budgetmaking. Central's use 
of incentive funds and program review findings, and Sunkist's establishment of 
an explicit program modification and discontinuance process all used the 
institution's formal authority structure to induce different kinds of behavior. 
In the case of Central and Frontier State, moreover, fiscal incentives for 
different kinds of behavior were also present. Furthermore, Midstate, Central, 
Heartland, and Vista all illustrate the importance of central administration 
encouraging the consistent use of performance information as an institution's 
official "language" of planning. Use of this language was instrumental ly 
encouraged and rewarded until those who used it began themselves to see its 
utility in improving their own units and departments. Through a gradual 
process of explicitly modifying language and behavior, changes gradually became 
internalized, creating an active, ongoing "information constituency" on each 
campus. 

A second theme, however, conditions the operation of this mechanism: the 
process of building participation must be carefully matched to the parameters 
of a particular institutional culture. Some cultures— such as that at Frontier 
State and Vista Community College— are suffiently distinctive that the entire 
shape of the utilization process must be tailored to fit it. In cases v'here 
the campus culture was already undergoing change, for example C<?ntral 
University and Midstate— both of which were in the process of becoming more 
centralized, the information utilization effort could be coupled with a larger, 
more legitimate change process. But in cases where the utilization effort was 
largely incongruent with existing campus culture, the result was to 
considerably temper success. For example at Southern State an initiative 
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perceived as being a "student affairs" effort that stressed coordinated effort 
was strongly at odds with both the decentralized tradition of academic 
decsisionmaking, and the faculty's disdain for "development" functions. 

A third theme is the use of an externally imposed requirement or an internal 
problem as a starting point for information utilization. Both can serve as 
powerful levers for mobilizing attention and action in the early stages of an 
improvement effort. But these cases also illustrate the many opportunities for 
the effort itself to become trapped by the parameters of the proDlem. At 
Southern State, attainment of "real" objectives ultimately deflated the 
perceived need for ongoing information utilization. With the onset of a new 
problem, a whole new set of response mechanisms will need to be developed. At 
Central University and Sunkist Community College, however, response to external 
mandates allowed administration to develop an institutional process with 
considerably greater legitimacy and decision latitude than would otherwise have 
been the case. While the short- term problem mobilized attention, the process 
that was built around it was gradually allowed to shape behavior. At Midstate 
and Vista Community College, in contrast, the lack of an urgent, generally 
recognized problem meant that similar processes took a good deal of time to get 
started. The implication for institutional practice is that use of a problem 
as an initial focus for a change effort is a useful device, but that if the 
effort remains problem or compliance-driven, it will not stick. 

Finally, all three of the above themes function within a fourth--the importance 
of sufficient time for incremental development and modification. A primary 
danger of time pressure is that it eliminates the subtle consensus building of 
the first mechanism. Building an "information constituency" at Coldspring 
State was impossible overnight. At Sunkist, moreover, initial compliance with 
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the results of the program review process was not because of its percieved 
utility or legitmacy; rather it was because such compliance was consistent with 
the institution's traditional "top-down" management style. Unfocused by the 
presence of a particular problem, both Visa Community College and Miastate 
showed no explicit "progress" in their programs a year after initiation. Both 
would have been declared "failures" if sufficient time had not been allowed for 
incremental changes in administrative behavior and attitudes to develop. 

If recent calls for information-based improvement are to be effectively 
implemented, a great deal more understanding will need to be built about what 
ipakes such efforts successful and unsuccessful in particular campus 
environments. Based on a few emerging cases, this review suggests some 
directions for such an inquiry. Because information-based change is complex, 
the inquiry itself will be complex, and must take into account many local 
variations in information type any campus culture. As more campuses experiment 
with such approaches, however, the database of experience will grow, and the 
general izeability of these results can be more thoroughly explored. 
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FORMAT FOR CAMPUS PROJECT CONSULTING REPORT 



NCHEMS/KELLOGG STUDENT OUTCOMES PROJECT 

Institutional Setting: 

The basic purpose of this section is to present salient features of 
the institution within which the particular campus project is housed. The 
attempt is to classify the institution along a few key decisionmaking 
dimensions so that we can make appropriate generalizations based on 
project results. The questions here are basically the same as the ones ^e 
used lasL year, so it may only be necessary to update your observations at 
that time. 

1. Institutional Profile : 

(This section will iiclude basic institutional "demographics" such as 
size, type, location, program array, and recent history; I include it 
here to indicate that we are certainly going to pay attention to this 
as part of Institutional Setting.) 

2. Institutional Mission : 

Does the institution have a clear sense of its own mission or central 
direction? Is this well communicated to or shared by mid-level 
administative staff and facuUy members? If you asked the question 
of mission to different kinds of people on campus, what kinds of 
answers do you think you would get from whom? Does the institution 
have a strong "personality"? In what ways does it see itself as 
unique or distinctive? 
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3. Decisionmaking 'Style ": 

Is the institution basically run in a centralized or a decentralized 
manner? How much authority do individual unit heads appear to have 
to make their own decisions and control their own resources? Are 
they allowed to take risks? In what areas? Are lines of authority 
clear or fuzzy? How do changes generally get made on this campus? 



4. Institutional Flexibility : 

Is the institution under notable external pressure— either financial 
or for accountability? Has it experienced significant retrenchment 
in the past five years? Is financial pressure a common topic of 
conversation among faculty and staff? Aside from finances, how much 
decision latitude does the institution appear to have in its 
relations with higher authority (state system, state coordinating 
boardj, governing board, etc.)* Have any of these external factors 
changed notably in the course of the project period? 

5. Information Use and Availability : 

What do you see as the role of information generally in making 
decisions on this campus? What are the main problems or advantages 
this institution has in getting information communicated to key 
decisionmakers? What are the incentives/rewards for use of 
information? Where is information generally located and who controls 
or has access to it? What is the attitude of the chief executive 
toward information? 
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6. Special Resources and Opportunities : 

Are there unique features of the campus environment which make it 
particularly effective in promoting the use of information in 
decisionmaking? For example, are there particular individual 
talents, computer resources, information resources, or analytical 
resources available which would probably not be available to an 
otherwise comparable campus? What would have to be transported over 
to an otherwise comparable campus to duplicate what happened on the 
project campus? 

Project Structure and Basic Strategy: 

The purpose of this section is to block out some major dimensions of 
information utilization efforts which I think are common to all campuses. 
Like the above, the attempt is to broadly classify particular efforts for 
ease of generalization. Seven cases are far too few to seriously propose 
a "matrix" of project dimensions by institutional dimensions, but the 
image of such a matrix has been of help to me in trying to sort through 
the many things that have happened. 

1. Unit Context : 

What is the administrative and political setting of the particular 
unit within which the campus project is housed? What is the campus 
project director's administrative and political position? How much 
and in what areas can he/she act independently to accomplish change? 
What approvals and alliances are necessary to get something moving? 
What difference do you think it makes to the project on campus that 
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it was housed in this particular unit? Has this affected the way the 
project has been viewed on campus? 



2. Project Centralizatio n; 

Where has the initiative for project activities rested? Has the 
project consisted of a centrally-directed set of activities— perhaps 
carried out by others in the institution— or has it rather cons.ited 
of many independent, loosely coordinated efforts? How reactive have 
project leaders been to events at lower levels? Has project 
direction shifted many times as new issues or opportunities arose, or 
was a basic, established agenda pretty much carried out from 
beginning to end? 



3, Target Units/Groups : 

At whom or what was the project primarily directed? Whose behavior 
were the project leaders trying to change? Was the attempted change 
intended to be institution-wide, or initially only directed at 
particular types of units or individuals? What level of change was 
aimed at—broad changes in institutional policies and procedures or 
more narrowly targeted changes such as curricular changes or changes 
in admissions policies, counseling procedures, etc.? How did these 
targets shift as the project developed? Where were the initial 
"successes" achieved, and how were these exploited or not exploited? 



4. Project "Independence" ; 

To what extent was the project structured and implemented as an 
independent, free-sta..-ing effort distinct from other institutional 
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structures and processes? Did the structure of the project 
"cross-cut" existing lines of authority and responsibility? Did it 
represent an innovative use of existing structures, processes, or 
lines of authority? If you asked most people at the institution, 
would they be able to distinguish "project" activities from 
"non-project" activities of the same general type? What difference 
do you think this made? 

Problem Orientation ; 

How strong was the link between the campus project and a specific, 
identified campus issue or problem? Did most people involved on 
campus see the project as an "information utilization pr^ ect" or 
rather as a "retention" project, a "planning/program review" project, 
an "accountability/compliance" project or whatever? What labels were 
actually used to describe the project on campus? How did the problem 
focus enhance or inhibit the project's general effectiveness in 
promoting better information use? 

Kellogg Roles : 

What is your assessment of the "symbolic" role of the project on 
campu'i? Has the recognition of being part of a multi-institutional 
project funded by a large foundation meant that these resources are 
having a greater (or different) impact than it institutional 
resources in the same amounts were directly applied to the problem? 
Has the institution made use of its "Kellogg Identity" in any ways 
that go beyond the project? 
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Project Activities: 

We will be receiving what I expect to be fairly complete 
documentation of individual campus project activities with the final 
reports required of each campus. What we need in addition is an external 
view of the effectiveness of these activities and some of their "flavor." 
The intent of the following is to get at some of these more interpretive 
questions rather than to serve as complete documentation for what 
happened. 

1. Ma jor Activities and Accomplishments : 

Briefly review the significant activities and accomplishments of the 
project. What accomplishments were direct consequences and what were 
indirect consequences of project activities? From your own point of 
view, which of these were the most significant and which were the 
mozt disappointing? How much of what occurred probably would have 
occurred had there been no Kellogg Project? 

2. Role of Project Oversight Committee : 

How strong a role did the project steering committee play in the 
project? Was it a necessary part of the project or could (or did) 
the project pretty much go on without it? What role, if any, did the 
committee in (a) heightening the visibility of the project on campus, 

(b) increasing communicction/coordination between key players, and 

(c) providing additional perspectives in terms of which to develop 
the implications of outcomes information? Based on this campus 
experience, how important is it to insisL on the presence of such a 
committee in an information utilization project? 



3. Consultant Role ; 

What is the best way of characterizing your role as a consultant at 
this institution? Have you been primarily a technical expert? A 
facilitator for identifying problems and bringing people together? 
An observer/researcher documenting what the institution is doing on 
its own? A monitor/evaluator from a funding source? How have these 
various roles developed or changed as the project has progressed? 
Which of them have been most successful? What is your best (detached 
and modest) guess as to what would have happened in the project had 
you not been around? Based on this campus experience, how important 
does external consultation appear to be in launching and carrying 
through an information utilization project? 

4. Project Timing : 

Was two years a realistic time frame to get something like this 
accomplished? If the project had been cut off at the end of the 
first year, what would there be to show for it? If the project had 
continued on for another two years with the same level of external 
lesources, would the impact have been greater or more permanent? Did 
the project seem to develop in terms of identifiable periods or 
phases? How distinct were these and what was their sequence and 
duration? Based on this campus experience, should we re-think the 
time-frame for a campus-wide information utilization project? 



Prognosis : 

Wh& 1o you expect these activities to look like a year from now? 
What efforts are currently being made to "reqularlize" project 
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activities and to assign project responsibilities to existing campus 
units, committees, and processes? What do you think their fate is 
likely to be? 

In Retrospect : 

Given the situation, the resources and the setting, was all done that 
could have been done? In retrospect, what actions or initiatives 
could realistically have been undertaken to enhance the effectiveness 
of the project? Looking back, were there any decisive incidents or 
moments when a new direction could have been taken? 
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DRAFT PROJECT ASSESSMENT INTERVIEW SCHEDULE 
NCHEMS/KELLOGG STUDENT OUTCOMES PROJECT 
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PR^3JECT GOALS: 

1. In one or two sentences, how would you describe the goals of 

the Kellogg Project at ? VJhat was the project trying 

to accomplish? [if not sure, give brief description of overall 
project goals and some examples of campus project activities,] 



2. How important do you feel these goals are on this campus at 
present? Did the project deal with issues or problems that 
were crucial auid visible, interesting but relatively unimportant, 
or of little interest or salience? 



PROJECT IMPACT: 

3. Thinking about the various activities undertaken through the 
project, what from your point of view have been the three most 
significant accomplishments? [May want to probe following areas: 
changes in academic policies, changes in curriculum or instruc- 
tional practices, changes in student service programs, changes 
in administrative procedures, or changes in knowledge about or 
attitudes toward students.] 



4. What have been the three greatest dissapointments or setbacks? 
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5. Can you identify any specific chamges in policies, procedures, 
or programs which occurred as a result of the project? Describe 
the nature of each change, where it was located, and how it 
came about. What further changes are planned as a result of the 
project? 



6. What new kinds of information about students have been generated 
or made available to you and others as a result of the project? 
How, if at all, have you used this information? Have you done 
anything differently as a result of having such information? 



PROJECT ENVIRONMENT: 

7. Thinking again about the major accomplishments and major dissapoint 
ments of the project which you identified earlier [remind if 
necessary]. What factors in the environment at _ do 

you feel were generally responsible for what occurred? [May 
want to probe following areas: Top-level administrative 
commitment, key people or resources, size and complexity of 
institution, salience of problem being addressed, particular 
strategy employed to disseminate or collect information, etc.] 



8. Briefly describe any important events over the last three years 
which you feel may have had an impact on the way the Kellogg 
Project developed on campus. 
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9. Who have been the greatest supporters of this project on campus 
[positions # responsibilities, reporting lines]? How have they 
provided support and how effective has it been in achieving the 
goals of the project? 



10. Can you think of anything else we ought to know about the way 
the project has developed here at ? 
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